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DETAILED ACTION 
Claim Rejections - 35 (JSC §102 

1 . The following is a quotation of the appropriate paragraphs of 35 U.S. C. 102 that form the 
basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(e) the invention was described in (1) an application for patent, published under section 122(b), by another filed 
in the United States before the invention by the applicant for patent or (2) a patent granted on an application for 
patent by another filed in the United States before the invention by the applicant for patent, except that an 
international application filed under the treaty defined in section 35 1(a) shall have the effects for purposes of this 
subsection of an application filed in the United States only if the international application designated the United 
States and was published under Article 21(2) of such treaty in the English language. 

2. Claims 1-27 are rejected under 35 U.S.C. 102(e) as being anticipated by Sato U.S. Patent 
No. 6,445,815. 

3. Claim 1: 

U.S. Pat. No. 6,445,815 to Sato teaches an augmented reality presentation apparatus 
(figures 6, 16, 18 and 20) for superimposing a virtual object in a real space (column 1, lines 6- 
23), characterized by comprising: 

objective viewpoint augmented reality presentation means (column 1, lines 51-63) for 
superimposing the virtual object viewed from a first viewpoint position, which differs from a 
player's viewpoint position, in the real space viewed from the first viewpoint position (figures 6, 
16 and 18, column 4, lines 64-67, column 5, lines 1-54, column 12, lines 1 1-50 and column 16, 
column 5, lines 1-61), wherein said objective viewpoint augmented reality presentation means 
includes 

First video sensing means for sensing a video of the real space viewed from the first 
viewpoint position (e.g., the three-dimensional position/posture sensor 101 mounted on the base 
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100, figures 6, 16 and 18, column 12, lines 1 1-50, column 16, column 5, lines 1-61 and column 
18, lines 5-15); 

First video generation means for generating a video of the virtual object viewed from the 
first viewpoint position (e.g., the position/posture module 201, column 12, lines 28-43 and the 
image generation module 300, column 12, lines 28-50, OR, the viewpoint position/posture 
estimation module 201 of the third embodiment, column 15, lines 28-44); 

First video composition means for composing an augmented reality video viewed from 
the first viewpoint position on the basis of the videos of the real space and the virtual object 
viewed from the first viewpoint position (The depth warping module 203 that warps the depth 
image at the viewpoint, column 13, lines 32-60), and 

Objective viewpoint video display means for displaying the augmented reality video 
obtained from said first video composition means (HMD 100 of figures 16 and 18, column 12, 
lines 11-50); 

wherein said apparatus further comprises: 

augmented reality presentation means for superimposing the virtual object viewed from 
the player's viewpoint position in the real space viewed from the player's viewpoint position 
(The Examiner interprets the player's viewpoint position as the observer's viewpoint position of 
Sato, column 18, lines 5-15, column 4, lines 64-67, column 5, lines 1-54, column 12, lines 1 1-50, 
column 13, lines 32-48, and column 15, lines 6-25); 

wherein said augmented reality presentation means includes 
second video sensing means for sensing a video of the real space viewed from the 
player's viewpoint position (e.g., virtual camera, column 13, lines 31-48); 
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second video generation means for generating a video of the virtual object viewed from 
the player's viewpoint position (e.g., the image generation module 300 of the third embodiment, 
column 34-43) ; 

second video composition means for composing an augmented reality video viewed from 
the player's viewpoint position on the basis of the videos of the real space and the virtual object 
viewed from the player's viewpoint position (e.g., depth warping module 203 of the third 
embodiment, column 16, lines 21-33 and the CG Tenderer 302 of the third embodiment, column 
16, lines 44-61); and 

display means for displaying to the player the augmented reality video viewed from the 
player's viewpoint position (e.g., column 14, lines 1-30, OR, figure 18, LCDs 103 for displaying 
a rendered image, column 15, lines 19-25). 

Claim 2: The apparatus according to claim 1, characterized in that said augmented 
reality presentation means further comprises: the second video sensing means for sensing a 
video of the real space viewed from said player 's viewpoint position; the second video 
generation means for generating a video of the virtual object viewed from said player 's 
viewpoint position; the second video composition means for compositing an augmented reality 
video viewed from said player 's viewpoint position on the basis of said videos of the real space 
and the virtual object viewed from said player 's viewpoint position; and the display means for 
displaying to the player the augmented reality video viewed from said player 's viewpoint 
position. 
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Claim 2 recites all the limitations of claim 1 and adds the limitation of "the second video 
sensing means," "the second video generation means," "the second video composition means," 
and "display means." The Sato reference teaches an augmented reality presentation means 
including a video sensing means such as HMD 100 with a three-dimensional position/posture 
sensor 101 for sensing a video of the virtual object (column 12, lines 12-27). The Sato reference 
also teaches a video generation means such as a depth image generation apparatus 200 for 
generating depth images in correspondence with the right and left viewpoints of the player 
(column 12-50). Sato further teaches a video composition means such as depth warping module 
203 and a CG renderer 302 for compositing an augmented reality video viewed from a player's 
viewpoint position (column 13, lines 32-60 and column 14, lines 1-29). Finally, Sato teaches 
display means such as LCDs 103 for displaying a rendered image (column 14, lines 1-29). 

Claim 3 : The apparatus according to claim 7, characterized in that said augmented 
reality presentation means further comprises: the second video generation means for generating 
a video of the virtual object viewed from said player 's viewpoint position; and the display means 
for displaying to the player the video of the virtual object viewed from said player 's viewpoint 
position on a display surface through which the player can visually see the real space. 

Claim 3 recites all the limitations of claim 1 and adds the limitation of "the second video 
generation means" and "the display means." The Sato reference teaches a video generation 
means such as a depth image generation apparatus 200 for generating depth images in 
correspondence with the right and left viewpoints of the player (column 12-50). The Sato 
reference further teaches display means such as LCDs 103 for displaying a rendered image 
(column 14, lines 1-29). 
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Claim 4: The apparatus according to claim I, characterized by further comprising 
information generation means for generating information that pertains to rendering of the 
virtual object, and in that said first video generation means and said second video generation 
means generate videos of the virtual object using the information that pertains to rendering of 
the virtual object. 

Claim 4 recites all the limitations of claim 1 and adds the limitation of "information 
generation means 5 ' and "generation means generate videos." The Sato reference teaches an 
information generation means such as the CG renderer 302 that renders an image and depth 
image of CG data received from the three-dimensional database 301 on the basis of the 
viewpoint position/posture information (column 14, lines 1-29) and the image generation module 
300 that generates an augmented reality image using the three-dimensional CG database in 
accordance with the distance to an object in the real world expressed by the depth image (column 
12, lines 44-50). 

Claim 5: The apparatus according to claim 4, characterized in that said information 
generation means generates information of an outer appearance of the virtual object and 
information of a position/posture of the virtual object as the information that pertains to 
rendering of the virtual object. 

Claim 5 recites all the limitations of claim 4 and adds the limitation of "generating 
information of an outer appearance of the virtual object and information of a position/posture of 
the virtual object." The Sato reference teaches that the CG renderer 302 renders an image and 
depth image of CG data received from the three-dimensional database 301 on the basis of the 
viewpoint position/posture information (column 14, lines 1-29) and that the image generation 
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module 300 generates an augmented reality image using the three-dimensional CG database in 
accordance with the distance to an object in the real world expressed by the depth image (column 
12, lines 44-50). 

Claim 6: The apparatus according to claim 1, characterized in that parameters of said 
first video sensing means are known, and said first video generation means generates the video 
of the virtual object viewed from said first viewpoint position in accordance with the known 
parameters. 

Claim 6 recites all the limitations of claim 1 and adds the limitation of "the known 
parameters/' The Sato reference teaches some known parameters of video sensing means such as 
the focal length of camera 102 (column 10, lines 62-67). 

Claim 7: The apparatus according to claim 1, characterized in that some of parameters 
of said first video sensing means are variable, said apparatus further comprises measurement 
means for measuring changes of the parameters, and said first video generation means generates 
the video of the virtual object viewed from said first viewpoint position in accordance with the 
parameters measured by said measurement means. 

Claim 7 recites all the limitations of claim 1 and adds the limitation of "variable 
parameters" and "measurement means." The Sato reference teaches the relative position/posture 
of the viewpoint of the camera 102 and the estimation module 201 to estimate a change in 
relative viewpoint position of the camera 102 viewed at different times on the basis of the time- 
sequentially extracted position/posture information of the camera 102 (column 5, lines 48-54). 
The Sato reference teaches measurement means in which the position/posture sensor 101 
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continuously outputs viewpoint position/posture information of the camera 102 along a time axis 
of the sensor 101 (column 8, lines 42-47). 

Claim 8: The apparatus according to claim 7, characterized in that the parameters of 
said first video sensing means measured by said measurement means include at least one of a 
viewpoint position/posture, and zoom ratio. 

Claim 8 recites all the limitations of claim 7 and adds the limitation of "position/posture 
information." The Sato reference teaches the relative position/posture of the viewpoint of the 
camera 102 and the estimation module 201 to estimate a change in relative viewpoint position of 
the camera 102 viewed at different times on the basis of the time-sequentially extracted 
position/posture information of the camera 102 (column 5, lines 48-54). 

Claim 9: The apparatus according to claim 1, characterized in that when a plurality of 
first video sensing means equivalent to said first video sensing means are present, said 
apparatus further comprises selection means for receiving a plurality of videos of the real space 
from said first viewpoint position from the plurality of first video sensing means, and outputting 
a video of the real space viewed from said first viewpoint position from one selected first video 
sensing means to said first video composition means, and said first video composition means 
generates a video of the virtual object viewed from said first viewpoint position using parameters 
of the first video sensing means selected by said selection means. 

Claim 9 recites all the limitations of claim 1 and adds the limitation of "selection means 
for receiving a plurality of first video sensing means." The Sato reference teaches two cameras 
102R and 102L for sensing a scene in front of them where image signals that represent an 
environment scene of a real space sensed by the respective cameras (column 5, lines 1-17) and 
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depth warping module 203 generate a depth image on the basis of an image from one camera 
depending on the purposes (column 7, lines 1-14). 

4. Claim 10: 

The claim 10 is a rephrasing of the claim 1 in a method form. The claim is rejected for 
the same reason as set forth in claim 1 . 

5. Claim 11: 

The method according to claim 10, characterized in that said augmented reality 
presentation step further comprises: the second video sensing step of sensing a video of the real 
space viewed from said player 's viewpoint position; the second video generation step of 
generating a video of the virtual object viewed from said player 's viewpoint position; the second 
video composition step of compositing an augmented reality video viewed from said player 's 
viewpoint position on the basis of said videos of the real space and the virtual object viewed 
from said player 's viewpoint position; and the display step of displaying to the player the 
augmented reality video viewed from said player 's viewpoint position. 

Claim 1 1 recites all the limitations of claim 10 and adds the limitation of "the second 
video sensing step", "the second video generation step", "the second video composition step" 
and "display step." The Sato reference teaches an augmented reality presentation step including a 
video sensing step for sensing a video of the virtual object in HMD 100 along with a three- 
dimensional position/posture sensor 101 (column 12, lines 12-27). The Sato reference also 
teaches a video generation step of generating depth images in correspondence with the right and 
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left viewpoints of the player in a depth image generation apparatus 200 (column 12-50). Sato 
further teaches a video composition step of compositing an augmented reality video viewed from 
a player's viewpoint position in depth warping module 203 and a CG renderer 302 (column 13, 
lines 32-60 and column 14, lines 1-29). Finally, Sato teaches display step of displaying a 
rendered image in LCDs 103 (column 14, lines 1-29). 

The Sato reference teaches in an alternative embodiment in figure 18 a program code of 
an augmented reality presentation step including a position/posture estimation module. 201, an 
image generation module 300, a depth-warping module 203 and a CG renderer 302. The Sato 
reference teaches the position/posture estimation module 201 that estimates movement of the 
viewpoint on the basis of images input from the cameras 102R and 102L (column 15, lines 38- 
44). Sato also teaches a depth warping module 203 that receives the transformation matrices 
representing three-dimensional motions and outputs depth images corresponding to the 
positions/postures of right and left viewpoints (column 16, lines 21-33). The Sato reference also 
teaches an alternative image generation module 300 that merges and outputs real images and 
images from the CG renderer 302 (column 16, lines 35-43). The Sato reference further teaches 
that the CG renderer 302 renders a real image and depth image of CG data received from a three- 
dimensional database 301 on the basis of the viewpoint position/posture information of the 
camera input from the position/posture estimation module (column 16, lines 34-67) and 
generates the real image to be displayed and that alternatively the CG renderer 302 receives the 
warped depth image from the depth warping module 203 and real image from the camera and 
renders a real image that has the depth of the depth image and renders and overlays a virtual 
image by using ordinal depth-keying technique (column 16, lines 52-61). 
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Claim 12: The apparatus according to claim 10, characterized in that said augmented 
reality presentation step further comprises: the second video generation step of generating a 
video of the virtual object viewed from said player 's viewpoint position; and the display step of 
displaying to the player the video of the virtual object viewed from said player 's viewpoint 
position on a display surface through which the player can visually see the real space. 

Claim 12 recites all the limitations of claim 10 and adds the limitation of "the second 
video generation step" and "the display step." The Sato reference teaches a video generation step 
of generating depth images in correspondence with the right and left viewpoints of the player in a 
depth image generation apparatus 200 (column 12-50). The Sato reference further teaches 
display step of displaying a rendered image in LCDs 103 (column 14, lines 1-29). 

Claim 13: The method according to claim 10, characterized by further comprising 
information generation step of generating information that pertains to rendering of the virtual 
object, and in that said first video generation step and said second video generation step 
generate videos of the virtual object using the information that pertains to rendering of the 
virtual object 

Claim 13 recites all the limitations of claim 10 and adds the limitation of "information 
generation step" and "first and second generation steps generate videos." The Sato reference 
teaches an information generation step in which the CG renderer 302 renders an image and depth 
image of CG data received from the three-dimensional database 301 on the basis of the 
viewpoint position/posture information (column 14, lines 1-29) and video generation steps in 
which the image generation module 300 generates an augmented reality image using the three- 
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dimensional CG database in accordance with the distance to an object in the real world expressed 
by the depth image (column 12, lines 44-50). 

Claim 14: The apparatus according to claim 13, characterized in that said information 
generation step includes the step of generating information of an outer appearance of the virtual 
object and information of a position/posture of the virtual object as the information that pertains 
to rendering of the virtual object 

Claim 14 recites all the limitations of claim 13 and adds the limitation of "generating 
information of an outer appearance of the virtual object and information of a position/posture of 
the virtual object." The Sato reference teaches an information generation step that the CG 
renderer 302 renders an image and depth image of CG data received from the three-dimensional 
database 301 on the basis of the viewpoint position/posture information (column 14, lines 1-29) 
and the image generation module 300 generates an augmented reality image using the three- 
dimensional CG database in accordance with the distance to an object in the real world expressed 
by the depth image (column 12, lines 44-50). 

Claim 15: The method according to claim 10, characterized in that parameters of said 
first video sensing means are known, and said first video generation step includes the step of 
generating the video of the virtual object viewed from said first viewpoint position in accordance 
with the known parameters. 

Claim 15 recites all the limitations of claim 10 and adds the limitation of "the known 
parameters." The Sato reference teaches the known parameters such as the focal length of 
camera 102 (column 10, lines 62-67). 
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Claim 16: The method according to claim 10, characterized in that some of parameters of 
means for sensing a video viewed from said first viewpoint position are variable, said method 
further comprises the measurement step of measuring changes of the parameters, and said first 
video generation step includes the step of generating the video of the virtual object viewed from 
said first viewpoint position in accordance with the parameters measured in the measurement 
step. 

Claim 16 recites all the limitations of claim 10 and adds the limitation of "variable 
parameters" and "measurement step." The Sato reference teaches the relative position/posture of 
the viewpoint of the camera 102 and the estimation module 201 to estimate a change in relative 
viewpoint position of the camera 102 viewed at different times on the basis of the time- 
sequentially extracted position/posture information of the camera 102 (column 5, lines 48-54). 
The Sato reference teaches measurement step in which the position/posture sensor 101 
continuously outputs viewpoint position/posture information of the camera 102 along a time axis 
of the sensor 101 (column 8, lines 42-47). 

Claim 17: The method according to claim 16, characterized in that the parameters of the 
means for sensing a video viewed from said first viewpoint position measured in the 
measurement step include at least one of a viewpoint position/posture, and zoon ration. 

Claim 17 recites all the limitations of claim 16 and adds the limitation of 
"position/posture information." The Sato reference teaches the relative position/posture of the 
viewpoint of the camera 102 and the estimation module 201 to estimate a change in relative 
viewpoint position of the camera 102 viewed at different times on the basis of the time- 
sequentially extracted position/posture information of the camera 102 (column 5, lines 48-54). 
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Claim 18: The method according to claim 10, characterized in that when a plurality of 
means for sensing a video viewed from said first viewpoint position are present, said method 
further comprises the selection step of receiving a plurality of videos of the real space viewed 
from a first viewpoint position from the plurality of means for sensing a video viewed from said 
first viewpoint position, and outputting the video of the real space viewed from a first viewpoint 
position input from one selected means for sensing a video of said first viewpoint position to 
means for compsiting a first viewpoint video, and said first video composition step includes the 
step of generating a video of the virtual object viewed from said first viewpoint position using 
parameters of the means for sensing a video viewed from a first viewpoint position selected in 
the selection step. 

Claim 18 recites all the limitations of claim 10 and adds the limitation of "selection step 
for receiving a plurality of videos." The Sato reference teaches a selection step in which a depth 
image is generated on the basis of an image from one camera (column 7, lines 1-14). 
6. Claim 19: 

The claim 19 encompasses the same scope of invention as that of claim 1 except 
additional claimed limitation of "a storage medium storing a program code. However, the Sato 
reference further discloses the claimed limitation of a storage medium that stores an image 
processing program, which is implemented on a computer and continuously presents three- 
dimensional images to an observer/player, storing a program code of an augmented reality, a 
depth estimation program code, a depth image generation program code, a position/posture 
information estimation program code, a warping program code and a program code of presenting 
to the observer three-dimensional images. The Sato reference implicitly teaches a program code 
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including the augmented reality presentation step of generating and presenting a virtual image in 
a real space in which an image of merged real objects and virtual objects are displayed in LCDs 
103 in an optical see-through head mounted device (column 1, lines 13-23). The Sato reference 
also implicitly teaches a position/posture estimation module 201 that outputs three-dimensional 
motions from a viewpoint position of the camera to right and left viewpoint positions of the 
player (column 12, lines 54-65). The Sato reference further teaches a depth image generation 
module 300 that uses position/posture information input from the position/posture estimation 
module 201 as that for CG rendering and which generates an augmented reality image using the 
three-dimensional CG database in accordance with the distance to an object in the real world 
expressed by the depth image and presents it on the LCDs 103 (column 12, lines 44-50) and a 
depth warping module 203 to inversely project a depth image ID acquired at a viewpoint having 
position/posture information into a space, and to re-project it onto the imaging plane of the 
virtual camera with the focal length of the virtual camera assumed at the viewpoint having an 
estimated position/posture value by the viewpoint position/posture module 201 (column 13, lines 
32-48). 

Claim 20: The medium according to claim 19, characterized in that the program code of 
the augmented reality presentation step further comprises: a program code of the second video 
sensing step of sensing a video of the real space viewed from said player 's viewpoint position; a 
program code of the second video generation step of generating a video of the virtual object 
viewed from said player 's viewpoint position; a program code of the second video composition 
step of compositing an augmented reality video viewed from said player 's viewpoint position on 
the basis of said videos of the real space and the virtual object viewed from said player 's 
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viewpoint position; and a program code of the display step of displaying to the player the 
augmented reality video viewed from said player 's viewpoint position. 

Claim 20 recites all the limitations of claim 19 and adds the limitation of "a program code 
of the second video sensing step", "a program code of the second video generation step", u a 
program code of the second video composition step" and "a program code of display step/' The 
Sato reference teaches a program code of an augmented reality presentation step including an 
alternative position/posture estimation module 201, an alternative image generation module 300, 
an alternative depth-warping module 203 and an alternative CG renderer 302 of the third 
embodiment. The Sato reference teaches the position/posture estimation module 201 that 
estimates movement of the viewpoint on the basis of images input from the cameras 102R and 
102L (column 15, lines 38-44). Sato also teaches a depth warping module 203 that receives the 
transformation matrices representing three-dimensional motions and outputs depth images 
corresponding to the positions/postures of right and left viewpoints (column 16, lines 21-33). 
The Sato reference also teaches an alternative image generation module 300 that merges and 
outputs real images and images from the CG renderer 302 (column 16, lines 35-43). The Sato 
reference further teaches that the CG renderer 302 renders a real image and depth image of CG 
data received from a three-dimensional database 301 on the basis of the viewpoint 
position/posture information of the camera input from the position/posture estimation module 
(column 16, lines 34-67) and generates the real image to be displayed and that alternatively the 
CG renderer 302 receives the warped depth image from the depth warping module 203 and real 
image from the camera and renders a real image that has the depth of the depth image and 
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renders and overlays a virtual image by using ordinal depth-keying technique (column 16, lines 
52-61). 

Claim 21 : The medium according to claim 19 \ characterized in that the program code of 
the augmented reality presentation step further comprises: a program code of the second video 
generation step of generating a video of the virtual object viewed from said player 's viewpoint 
position; and a program code of the display step of displaying to the player the video of the 
virtual object viewed from said player 's viewpoint position on a display surface through which 
the player can visually see the real space. 

Claim 21 recites all the limitations of claim 19 and adds the limitation of "a program code 
of the second video generation step" and "a program of the display step." The Sato reference also 
teaches an alternative image generation module 300 that merges and outputs real images and 
images from the CG Tenderer 302 (column 16, lines 35-43). The Sato reference further teaches 
that the CG renderer 302 renders a real image and depth image of CG data received from a three- 
dimensional database 301 on the basis of the viewpoint position/posture information of the 
camera input from the position/posture estimation module (column 16, lines 34-67) and 
generates the real image to be displayed and that alternatively the CG renderer 302 receives the 
warped depth image from the depth warping module 203 and real image from the camera and 
renders a real image that has the depth of the depth image and renders and overlays a virtual 
image by using ordinal depth-keying technique (column 16, lines 52-61). 

Claim 22: The medium according to claim 19, characterized by further comprising a 
program code of the information generation step of generating information that pertains to 
rendering of the virtual object, and in that in the program code of said first video generation step 
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and said second video generation step, videos of the virtual object are generated using the 
information that pertains to rendering of the virtual object 

Claim 22 recites all the limitations of claim 19 and adds the limitation of "a program code 
of the information generation step" and "videos of virtual objects are generated." The Sato 
reference teaches in figures 18 and 20 that the CG Tenderer 302 renders an image and depth 
image of CG data received from the three-dimensional database 301 on the basis of the 
viewpoint position/posture information and the image generation module 300 generates an 
augmented reality image using the three-dimensional CG database in accordance with the 
distance to an object in the real world expressed by the depth image. 

Claim 23: The medium according to claim 22, characterized in that the program code of 
said information generation step includes the step of generating information of an outer 
appearance of the virtual object and information of a position/posture of the virtual object as the 
information that pertains to rendering of the virtual object 

Claim 23 recites all the limitations of claim 22 and adds the limitation of "generating 
information of an outer appearance of the virtual object and information of a position/posture of 
the virtual object " The Sato reference teaches in figure 18 and 20 an information generation step 
that the CG renderer 302 renders an image and depth image of CG data received from the three- 
dimensional database 301 on the basis of the viewpoint position/posture information and the 
image generation module 300 generates an augmented reality image using the three-dimensional 
CG database in accordance with the distance to an object in the real world expressed by the 
depth image. 
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Claim 24: The medium according to claim 19, characterized in that parameters of means 
for sensing said first viewpoint video are known, and the program code of said first video 
generation step includes the step of generating the video of the virtual object viewed from said 
first viewpoint position in accordance with the known parameters. 

Claim 24 recites all the limitations of claim 19 and adds the limitation of "the known 
parameters." The Sato reference teaches in particular the known parameters such as the focal 
length of each camera 102 (column 16, lines 21-33). 

Claim 25: The medium according to claim 19, characterized in that some of parameters 
of means for sensing a video viewed from said first viewpoint position are variable, the program 
code of said medium further comprises the measurement step of measuring changes of the 
parameters, and the program code of said first video generation step includes the step of 
generating the video of the virtual object viewed from said first viewpoint position in accordance 
with the parameters measured in the measurement step. 

Claim 25 recites all the limitations of claim 19 and adds the limitation of "variable 
parameters" and "measurement step." The Sato reference teaches measurement step in which the 
position/posture estimation module 201 estimates movement of the viewpoint on the basis of 
images input from the cameras 102R and 102L by tracking changes in coordinate value of 
feature points and the position on the real space of which is known (column 15, lines 38-67). 

Claim 26: The medium according to claim 25, characterized in that the parameters of the 
means of sensing a video viewed from said first viewpoint position measured in the measurement 
step include at least one of a viewpoint position/posture, and zoon ration. 
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Claim 26 recites all the limitations of claim 25 and adds the limitation of 
tc position/posture information." The Sato reference teaches in figure 18 a head mounted 
position/posture sensor 101 and camera 102. The Sato reference further teaches various schemes 
for estimating viewpoint position/posture on the basis of image input from the cameras 102R and 
102L (column 16, lines 38-67). 

Claim 27: The medium according to claim 19, characterized in that when a plurality of 
means for sensing a video viewed from said first viewpoint position are present, said medium 
further comprises a program code of the selection step of receiving a plurality of videos of the 
real space viewed from a first viewpoint position from the plurality of means for sensing a video 
viewed from said first viewpoint position, and outputting the video of the real space viewed from 
a first viewpoint position input from one selected means for sensing a video of said first 
viewpoint position to means for compsiting a first viewpoint video, and the program code of said 
first video composition step includes the step of generating a video of the virtual object viewed 
from said first viewpoint position using parameters of the means for sensing a video viewed from 
a first viewpoint position selected in the selection step. 

Claim 27 recites all the limitations of claim 19 and adds the limitation of "selection step 
for receiving a plurality of videos." The Sato reference teaches in figure 18 a depth estimation 
module 202, a viewpoint position/posture estimation module, a depth warping module and an 
image generation module. The Sato reference further teaches a selection step in which a depth 
image is generated on the basis of an image from one camera (see also column 7, lines 1-14). 



Claim Rejections - 35 USC §103 
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The following is a quotation of 35 U.S. C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

Claim 28-30 are rejected under 35 U.S.C. 103(a) as being unpatentable over Sato U.S. 
Patent No. 6,445,815. 

Referring to claims 28-30, the Sato reference discloses an augmented reality presentation 
system that generates and presents a virtual image in a real space (see figures 18-20 of Sato). The 
Sato reference teaches in figure 20 a video composition means that outputs display image for 
displaying on 103R and 103L. However, the reference is silent on a printing means in connection 
to the augmented reality presentation apparatus. It is common that a computer system has a 
printing means attached to them. Therefore, it would have been obvious to one having ordinary 
skill in the art to have incorporated a printing means in the augmented reality presentation 
apparatus of Sato because such construction is rather conventional. A person of ordinary skill in 
the art would be motivated to have incorporated a printing means to the Sato's augmented reality 
presentation apparatus to further provide a paper copy of still images of the real images that have 
been displayed on the display devices of Sato. 



Remarks 

7. Applicant's arguments, filed 02/28/2003, paper number 4, have been fully considered but 
they are not deemed to be persuasive. 

8. Applicant argues in essence with respect to claim 1 and similar claims that: 
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"By these features, an augmented reality video (i.e., a virtual object superimposed in a 
real space) can be displayed according to both a player's viewpoint and a viewpoint that 
differs from the player's viewpoint (for example, according to the viewpoint of camera 
103 in Fig. 1). Applicants submit that the cited art fails to disclose or suggest at least 
these features." 
This is not found persuasive for the following reasons: 

(a) The Examiner interprets "a player's viewpoint" as a viewpoint from a player who 
wears an HMD on his head (Applicants' specification, page 1 1, lines 4-7). 

In figures 6, 16 and 18, column 4, lines 64-67 and column 5, lines 1-54 of Sato, it is 
stated that "The base 100 has the two cameras 102R and 102L for stereoscopically sensing a 
scene in front of them. Image signals Ir and II that represent an environmental scene of a real 
space sensed by the respective cameras are sent to. . . a module 201 for estimating the relative 
position/posture of the viewpoint of the camera 102R.. ." and in figure 16 and column 12, lines 
14-50 of Sato, it is stated that "First, the second embodiment requires a depth image at each 
viewpoint of the camera 102R. Second, depth images must be generated in correspondence with 
the right and left viewpoints of the observer. . . " 

In column 15, lines 1 1-25, it is stated "Upon comparing the constructing elements of the 
system of the third embodiment in FIG. 18 with those of the system of the second embodiment 
shown in FIG. 16, the former system is different from the latter one in that the former system has 
no head mounted position/posture sensor 101 and a viewpoint position position/posture 
estimation module 201 can estimate movement of the viewpoint from an image acquired by one 
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camera 102. Since the third embodiment uses a video see-through HMD, the arrangement of an 
image generation module is also different from the second embodiment. 

In column 16, lines 44-61, column 13, lines 31-48, and column 18, lines 5-15 of Sato, it 
is stated that "when a virtual camera equivalent to the viewpoint of the observer (player) is 
assumed, the operation of the depth warping module 203 of the second embodiment is to 
inversely project a depth image aquired at a viewpoint having position/posture information into a 
space," and " Both three-dimensional position/posture sensor and camera may be used together in 
a modification " to the second and third embodiments. 

As applied to the present application, Sato fulfills the amended claim limitation of "a 
player's viewpoint." 

(b) The Examiner interprets "a viewpoint (a first viewpoint position as recited in claim 1) 
that differs from the player's viewpoint" as an objective viewpoint that is fixed at a 
predetermined position and posture (Applicant's specification, page 11, lines 25-26, page 12, 
lines 1-4). In figure 6 and column 4, lines 64-67 and column 5, lines 1-54 of Sato, it is stated that 
"The three-dimensional position/posture sensor 101 is mounted on the base 100. . ." and in figure 
16 and column 12, lines 14-50 of Sato, it is stated that "the principle of the embodiment is 
applied to an optical see-through augmented reality presentation system. . . In order to accurately 
detect the viewpoint position, a three dimensional position/posture sensor 101 is provided to the 
HMD 100 ". 

Moreover, in column 18, lines 5-15 of Sato, it is stated "The viewpoint position/posture 
estimation module in the first or second embodiment uses information from the three- 
dimensional position/posture sensor, and that in the third embodiment uses image information 
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from the cameras. However, these embodiments can be practiced using either scheme. Further. 
both thre e-dimensional position/posture sensor and camera may be used together in a 
modification. Sato clearly teaches that the second embodiment can use information from the 
camera and the third embodiment can use image information from the three-dimensional 
position/posture sensor. In another word, the sensor and camera can be used interchangeably. 

As applied to the present application, Sato fulfills the claimed limitation of an objective 
viewpoint (a first viewpoint position of claim 1) that differs from the player's viewpoint that can 
be obtained both from the sensor/camera. The Examiner notes that the objective viewpoint from 
sensor/camera is different from the player's viewpoint obtained from camera 102R/102L or 102. 

(c) In summary, Sato clearly teaches an augmented reality video that can be displayed 
according to both a player's viewpoint and a viewpoint which differs from the player's viewpoint 
(for example, according to the viewpoint of camera 102R and 102L in Fig. 16) and therefore 
Sato fulfills claim 1 as amended. 



9. Applicant argues in essence that: 

"Applicants submit that Sato merely discloses a configuration for generating an 
augmented image according to a player's viewpoint, and it fails to disclose or suggest at 
least the features of Claim 1 of displaying an augmented reality video viewed from a 
player's position and viewed from an objective viewpoint." 

This is not found persuasive because of the following reasons: 

The Examiner interprets "a viewpoint (a first viewpoint position as recited in claim 1) 

that differs from the player's viewpoint" as an objective viewpoint that is fixed at a 
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predetermined position and posture (Applicant's specification, page 1 1, lines 25-26, page 12, 
lines 1-4). In figure 6 and column 4, lines 64-67 and column 5, lines 1-54 of Sato, it is stated that 
"The three-dimensional position/posture sensor 101 is mounted on the base 100. . ." and in figure 
16 and column 12, lines 14-50 of Sato, it is stated that "the principle of the embodiment is 
applied to an optical see-through augmented reality presentation system. . . In order to accurately 
detect the viewpoint position, a three dimensional position/posture sensor 101 is provided to the 
HMD 100 ". 

Moreover, in column 18, lines 5-15 of Sato, it is stated "The viewpoint position/posture 
estimation module in the first or second embodiment uses information from the three- 
dimensional position/posture sensor, and that in the third embodiment uses image information 
from the cameras. However these embodiments can be practiced using either scheme. Further, 
both three-dimensional position/posture sensor and camera may be used together in a 
modification Sato clearly teaches that the second embodiment can use information from the 
camera and the third embodiment can use image information from the three-dimensional 
position/posture sensor. In another word, the sensor and camera can be used interchangeably. 

As applied to the present application, Sato fulfills the claimed limitation of an objective 
viewpoint (a first viewpoint position of claim 1) that differs from the player's viewpoint that can 
be obtained both from the sensor/camera. The Examiner notes that the objective viewpoint from 
sensor/camera is different from the player's viewpoint obtained from camera 102R/102L or 102. 



Conclusion 
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10. THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1 .136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1 . 136(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the mailing 
date of this final action. 

1 1 . Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Jin-Cheng Wang whose telephone number is (703) 605-1213. 
The examiner can normally be reached on 8:00 AM - 4:30 PM. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Mike Razavi can be reached on (703) 305-4713. The fax phone numbers for the 
organization where this application or proceeding is assigned are (703) 308-6606 for regular 
communications and (703) 308-6606 for After Final communications. 

Any inquiry of a general nature or relating to the status of this application or proceeding 
should be directed to the receptionist whose telephone number is (703) 395-3900. 
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